La modélisation et le contrôle des services BigData : application à la performance et la fiabilité de MapReduce

The amount of raw data produced by everything from our mobile phones, tablets, computers to our smart watches brings novel challenges in data storage and analysis. Many solutions have arisen in the industry to treat these large quantities of raw data, the most popular being the MapReduce framework. However, while the deployment complexity of such computing systems is steadily increasing, continuous availability and fast response times are still the expected norm. Furthermore, with the advent of virtualization and cloud solutions, the environments where these systems need to run is becoming more and more dynamic. Therefore ensuring performance and dependability constraints of a MapReduce service still poses significant challenges. In this thesis we address this problematic of guaranteeing the performance and availability of MapReduce based cloud services, taking an approach based on control theory. We develop the first dynamic models of a MapReduce service running a concurrent workload. Furthermore, we develop several control laws to ensure different quality of service objectives. First, classical feedback and feedforward controllers are developed to guarantee service performance. To further adapt our controllers to the cloud, such as minimizing the number of reconfigurations and costs, a novel event-based control architecture is introduced for performance management. Finally we develop the optimal control architecture MR-Ctrl, which is the first solution to provide guarantees in terms of both performance and dependability for MapReduce systems, meanwhile keeping cost at a minimum. All the modeling and control approaches are evaluated both in simulation and experimentally using MRBS, a comprehensive benchmark suite for evaluating the performance and dependability of MapReduce systems. Validation experiments were run in a real 60 node Hadoop MapReduce cluster, running a data intensive Business Intelligence workload. Our experiments show that the proposed techniques can successfully guarantee performance and dependability constraints.Le grand volume de données généré par nos téléphones mobiles, tablettes, ordinateurs, ainsi que nos montres connectées présente un défi pour le stockage et l'analyse. De nombreuses solutions ont émergées dans l'industrie pour traiter cette grande quantité de données, la plus populaire d'entre elles est MapReduce. Bien que la complexité de déploiement des systèmes informatiques soit en constante augmentation, la disponibilité permanente et la rapidité du temps de réponse sont toujours une priorité. En outre, avec l'émergence des solutions de virtualisation et du cloud, les environnements de fonctionnement sont devenus de plus en plus dynamiques. Par conséquent, assurer les contraintes de performance et de fiabilité d'un service MapReduce pose un véritable challenge. Dans cette thèse, les problématiques de garantie de la performance et de la disponibilité de services de cloud MapReduce sont abordées en utilisant une approche basée sur la théorie du contrôle. Pour commencer, plusieurs modèles dynamiques d'un service MapReduce exécutant simultanément de multiples tâches sont introduits. Par la suite, plusieurs lois de contrôle assurant les différents objectifs de qualités de service sont synthétisées. Des contrôleurs classiques par retour de sortie avec feedforward garantissant les performances de service ont d'abord été développés. Afin d'adapter nos contrôleurs au cloud, tout en minimisant le nombre de reconfigurations et les coûts, une nouvelle architecture de contrôle événementiel a été mise en œuvre. Finalement, l'architecture de contrôle optimal MR-Ctrl a été développée. C'est la première solution à fournir aux systèmes MapReduce des garanties en termes de performances et de disponibilité, tout en minimisant le coût. Les approches de modélisation et de contrôle ont été évaluées à la fois en simulation, et en expérimentation sous MRBS, qui est une suite de tests complète pour évaluer la performance et la fiabilité des systèmes MapReduce. Les tests ont été effectuées en ligne sur un cluster MapReduce de 60 nœuds exécutant une tâche de calcul intensive de type Business Intelligence. Nos expériences montrent que le contrôle ainsi conçu, peut garantir les contraintes de performance et de disponibilité

Modeling and Control of MapReduce Systems

posterInternational audienceSystems based on the MapReduce programming model are emerging as a central tool for deploying jobs that process large datasets in parallel. However the configuration of MapReduce systems is a complex process and at the moments it's left up to the user. These ad-hoc configuration methods make it difficult for small companies to take advantage of the growth of cloud computing solutions that provide resources as a service. Furthermore, the definition of SLAs becomes a complicated process for the user and the service provider as well. We propose a control theoretical approach to solving these problems. This implies the development of a general model that captures the dynamics of MapReduce systems. Finally, we intend to provide novel control methods that ease the configuration process and guarantee service level objectives such as constraints on system performance (execution times) and dependability (latency, availability) while optimizing resource consumption

Adaptive Optimal Control of MapReduce Performance, Availability and Costs

International audienceMapReduce is a popular programming model for distributed data processing and Big Data applications running on clouds. Extensive research has been conducted either to improve the dependability or to increase performance of MapReduce, ranging from adaptive and on-demand fault-tolerance solutions, adaptive task scheduling techniques to optimized job execution mechanisms. This paper investigates an optimization-based solution to control MapReduce systems in order to provide guarantees in terms of both performance and availability while reducing utilization costs. We follow a control theoretical approach for MapReduce cluster scaling and admission control. Moreover, we aim to be robust to changes in MapRe-duce and in it's environment by adapting the controller online to those changes. This paper highlights the major challenges of combining system adaptation and optimal control to take the best of both approaches. CCS Concepts • Networks → Cloud computing; • Software and its engineering → Software configuration management and version control systems; • Computer systems organization → Dependable and fault-tolerant systems and networks

Adaptive Modelling and Control in Distributed Systems

International audienceCompanies have growing amounts of data to store and to process. In response to these new processing challenges, Google developed MapReduce, a parallel programming paradigm which is becoming the major tool for BigData treatment. Even if MapReduce is used by most IT companies, ensuring its performances while minimizing costs is a real challenge requiring a high level of expertise. Modelling and control of MapReduce have been developed in the last years, however there are still many problems caused by the software's high variability. To tackle the latter issue, this paper proposes an on-line model estimation algorithm for MapReduce systems. An adaptive control strategy is developed and implemented to guarantee response time performances under a concurrent workload while minimizing resource use. Results have been validated using a 40 nodes MapReduce cluster under a data intensive Business Intelligence workload running on Grid5000, a French national cloud. The experiments show that the adaptive control algorithm manages to guarantee performances and low costs even in a highly variable environment

Feedback Autonomic Provisioning for Guaranteeing Performance in MapReduce Systems

International audienceCompanies have a fast growing amounts of data to process and store, a data explosion is happening next to us. Currentlyone of the most common approaches to treat these vast data quantities are based on the MapReduce parallel programming paradigm.While its use is widespread in the industry, ensuring performance constraints, while at the same time minimizing costs, still providesconsiderable challenges. We propose a coarse grained control theoretical approach, based on techniques that have already provedtheir usefulness in the control community. We introduce the first algorithm to create dynamic models for Big Data MapReduce systems,running a concurrent workload. Furthermore we identify two important control use cases: relaxed performance - minimal resourceand strict performance. For the first case we develop two feedback control mechanism. A classical feedback controller and an evenbasedfeedback, that minimises the number of cluster reconfigurations as well. Moreover, to address strict performance requirements afeedforward predictive controller that efficiently suppresses the effects of large workload size variations is developed. All the controllersare validated online in a benchmark running in a real 60 node MapReduce cluster, using a data intensive Business Intelligenceworkload. Our experiments demonstrate the success of the control strategies employed in assuring service time constraints

Application du contrôle pour garantir la performance des systèmes Big Data

International audienceNous sommes à l'aube d'une énorme explosion de données et la quantité à traiter par les entreprises est de plus en plus grande. Pour faire face à ce chalenge, Google a développé MapReduce, un modèle de programmation parallèle qui est en train de devenir l'outil de facto pour l'analyse des systèmes Big Data. Bien que dans une certaine mesure son utilisation est déjà très répandue dans l'industrie, garantir les performances d'un système aussi complexe pose de grands problèmes et sa gestion nécessite un haut niveau d'expertise. Cet article répond à ces défis en proposant le premier système autonome qui garantit des contraintes de temps de réponse pour une charge de travail MapReduce simultanée. Nous développons le premier modèle dynamique d'une grappe MapRe- duce. De plus, un contrôle en boucle fermée est conçu et implémenté pour garantir un temps de réponse donné. Un contrôle d'anticipation de type ""feedforward"" est également rajouté pour amé- liorer la réponse du système en présence de perturbations, en l'occurrence, la variation du nombre de clients. L'approche est validée en ligne sur une grappe MapReduce avec 40 nœuds utilisant une charge de travail intensive de type Business Intelligence. Nos expériences montrent que le contrôle ainsi conçu peut garantir les contraintes de temps de réponse

Cost Function based Event Triggered Model Predictive Controllers - Application to Big Data Cloud Services

International audienceHigh rate cluster reconfigurations is a costly issue in Big Data Cloud services. Current control solutions manage to scale the cluster according to the workload, however they do not try to minimize the number of system reconfigurations. Event-based control is known to reduce the number of control updates typically by waiting for the system states to degrade below a given threshold before reacting. However, computer science systems often have exogenous inputs (such as clients connections) with delayed impacts that can enable to anticipate states degradation. In this paper, a novel event-triggered approach is proposed. This triggering mechanism relies on a Model Predictive Controller and is defined upon the value of the optimal cost function instead of the state or output error. This controller reduces the number of control changes, in the normal operation mode, through constraints in the MPC formulation but also assures a very reactive behavior to changes of exogenous inputs. This novel control approach is evaluated using a model validated on a real Big Data system. The controller efficiently scales the cluster according to specifications, meanwhile reducing its reconfigurations

Towards Control of MapReduce Performance and Availability

International audienceMapReduce is a popular programming model for distributed data processing and Big Data applications. Extensive research has been conducted either to improve the dependability or to increase performance of MapReduce, ranging from adaptive and on-demand fault-tolerance solutions, adaptive task scheduling techniques to optimized job execution mechanisms. This paper investigates a novel solution that controls MapReduce systems and provides guarantees in terms of both performance and availability, while reducing utilization costs. We follow a control theoretic approach for MapReduce cluster scaling and admission control. Preliminary results based on a simulation environment, previously validated on a real MapReduce cluster, show the effectiveness of the proposed control solutions for a Hadoop MapReduce cluster

Modeling and control of cloud services : application to MapReduce performance and dependability

Le grand volume de données généré par nos téléphones mobiles, tablettes, ordinateurs, ainsi que nos montres connectées présente un défi pour le stockage et l'analyse. De nombreuses solutions ont émergées dans l'industrie pour traiter cette grande quantité de données, la plus populaire d'entre elles est MapReduce. Bien que la complexité de déploiement des systèmes informatiques soit en constante augmentation, la disponibilité permanente et la rapidité du temps de réponse sont toujours une priorité. En outre, avec l'émergence des solutions de virtualisation et du cloud, les environnements de fonctionnement sont devenus de plus en plus dynamiques. Par conséquent, assurer les contraintes de performance et de fiabilité d'un service MapReduce pose un véritable challenge. Dans cette thèse, les problématiques de garantie de la performance et de la disponibilité de services de cloud MapReduce sont abordées en utilisant une approche basée sur la théorie du contrôle. Pour commencer, plusieurs modèles dynamiques d'un service MapReduce exécutant simultanément de multiples tâches sont introduits. Par la suite, plusieurs lois de contrôle assurant les différents objectifs de qualités de service sont synthétisées. Des contrôleurs classiques par retour de sortie avec feedforward garantissant les performances de service ont d'abord été développés. Afin d'adapter nos contrôleurs au cloud, tout en minimisant le nombre de reconfigurations et les coûts, une nouvelle architecture de contrôle événementiel a été mise en œuvre. Finalement, l'architecture de contrôle optimal MR-Ctrl a été développée. C'est la première solution à fournir aux systèmes MapReduce des garanties en termes de performances et de disponibilité, tout en minimisant le coût. Les approches de modélisation et de contrôle ont été évaluées à la fois en simulation, et en expérimentation sous MRBS, qui est une suite de tests complète pour évaluer la performance et la fiabilité des systèmes MapReduce. Les tests ont été effectuées en ligne sur un cluster MapReduce de 60 nœuds exécutant une tâche de calcul intensive de type Business Intelligence. Nos expériences montrent que le contrôle ainsi conçu, peut garantir les contraintes de performance et de disponibilité.The amount of raw data produced by everything from our mobile phones, tablets, computers to our smart watches brings novel challenges in data storage and analysis. Many solutions have arisen in the industry to treat these large quantities of raw data, the most popular being the MapReduce framework. However, while the deployment complexity of such computing systems is steadily increasing, continuous availability and fast response times are still the expected norm. Furthermore, with the advent of virtualization and cloud solutions, the environments where these systems need to run is becoming more and more dynamic. Therefore ensuring performance and dependability constraints of a MapReduce service still poses significant challenges. In this thesis we address this problematic of guaranteeing the performance and availability of MapReduce based cloud services, taking an approach based on control theory. We develop the first dynamic models of a MapReduce service running a concurrent workload. Furthermore, we develop several control laws to ensure different quality of service objectives. First, classical feedback and feedforward controllers are developed to guarantee service performance. To further adapt our controllers to the cloud, such as minimizing the number of reconfigurations and costs, a novel event-based control architecture is introduced for performance management. Finally we develop the optimal control architecture MR-Ctrl, which is the first solution to provide guarantees in terms of both performance and dependability for MapReduce systems, meanwhile keeping cost at a minimum. All the modeling and control approaches are evaluated both in simulation and experimentally using MRBS, a comprehensive benchmark suite for evaluating the performance and dependability of MapReduce systems. Validation experiments were run in a real 60 node Hadoop MapReduce cluster, running a data intensive Business Intelligence workload. Our experiments show that the proposed techniques can successfully guarantee performance and dependability constraints

Control-Theoretical Software Adaptation: A Systematic Literature Review

Modern software applications are subject to uncertain operating conditions, such as dynamics in the availability of services and variations of system goals. Consequently, runtime changes cannot be ignored, but often cannot be predicted at design time. Control theory has been identified as a principled way of addressing runtime changes and it has been applied successfully to modify the structure and behavior of software applications. Most of the times, however, the adaptation targeted the resources that the software has available for execution (CPU, storage, etc.) more than the software application itself. This paper investigates the research efforts that have been conducted to make software adaptable by modifying the software rather than the resource allocated to its execution. This paper aims to identify: the focus of research on control-theoretical software adaptation; how software is modeled and what control mechanisms are used to adapt software; what software qualities and controller guarantees are considered. To that end, we performed a systematic literature review in which we extracted data from 42 primary studies selected from 1512 papers that resulted from an automatic search. The results of our investigation show that even though the behavior of software is considered non-linear, research efforts use linear models to represent it, with some success. Also, the control strategies that are most often considered are classic control, mostly in the form of Proportional and Integral controllers, and Model Predictive Control. The paper also discusses sensing and actuating strategies that are prominent for software adaptation and the (often neglected) proof of formal properties. Finally, we distill open challenges for control-theoretical software adaptation.status: accepte